{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Iris TensorFlow Keras\n",
    "\n",
    "In this notebook, we consider the well known iris classification problem of predicting iris species based on sepal and petal size.  We leverage Determined to train and tune a TensorFlow Keras model on TensorFlow's publicly available iris [training](http://download.tensorflow.org/data/iris_training.csv) and [test](http://download.tensorflow.org/data/iris_test.csv) datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test importing Determined.  If Determined is properly installed, you will see no output.\n",
    "import determined as det"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace with the IP address of the Determined master.\n",
    "determined_master = '<master IP>'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's explore the components of a Determined workflow in which we develop, train, and tune an iris species classifier.  In the model directory, we see files defining the dataset, model, and experiments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aside from this notebook, the directory contains:\n",
    "- `data.py`: The Determined data loader interface implementation\n",
    "- `model_def.py`: The TensorFlow Keras model definition exposed to Determined\n",
    "- `__init__.py`: The entrypoint for Determined to resolve the data loader interface and experiment `Trial` interface\n",
    "- `*.yaml` configuration files that each define an experiment\n",
    "\n",
    "Let's take a closer look at these files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### data.py\n",
    "<a id='data_py'></a>\n",
    "The required `make_data_loaders` function in `data.py` specifies the training and validation datasets to use in our experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat data.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### model_def.py\n",
    "Now let's view the implementation of Determined's `TFKerasTrial` interface, where we specify the classification model architecture. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat -n model_def.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \\_\\_init\\_\\_.py\n",
    "Given that a Determined experiment definition is a Python package, `__init__.py` is the entrypoint that exposes the trial implementation and data loader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat __init__.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### const.yaml\n",
    "This experiment configuration file specifies a single-trial experiment in which we train our classifier with fixed hyperparameters.  The top-level sections specify the experiment metadata, hyperparameters, and hyperparameter search algorithm to use:\n",
    "- `description`: A short description of the experiment\n",
    "- `data`: A section for the user to provide arbitrary key-value pairs referenceable at runtime.  Here we specify where the training and validation datasets reside.\n",
    "- `hyperparameters`: Constant hyperparameter values injected into the trial at runtime\n",
    "- `searcher`: The hyperparameter search algorithm for the experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat const.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run this experiment!  We submit the experiment configuration and model directory to the Determined master:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!det -m {determined_master} experiment create const.yaml ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the experiment completes (which may take a few minutes if Determined agents have to start up), we can see on the experiment detail page that we achieve 90+% accuracy on the test dataset.\n",
    "\n",
    "\n",
    "Next, let's run an experiment with the same model definition, but this time we leverage Determined's adaptive hyperparameter search to efficiently determine the hyperparameter values that yield the best-performing model.  We specify the hyperparameters as ranges instead of fixed values, and the `adaptive_simple` searcher to explore the hyperparameter space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat adaptive.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's submit this model tuning experiment to Determined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!det -m {determined_master} experiment create adaptive.yaml ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the experiment detail page, we see the best categorical accuracy that Determined's adaptive search achieves over time.  When the experiment finishes, we find that we reach 100% accuracy on the 30 test set examples, an improvement over the results of the fixed hyperparameter experiment.  We can drill in to the best-performing trial and view the associated hyperparameter values.  We can also access the saved checkpoint of our best-performing model and load it for real-time or batch inference as described in the [TensorFlow Keras documentation](https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/keras/models/load_model)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
